Where Should We Begin? A Low-Level Exploration of Weight Initialization Impact on Quantized Behaviour of Deep Neural Networks

With the proliferation of deep convolutional neural network (CNN) algorithms for mobile processing, limited precision quantization has become an essential tool CNN efficiency. Consequently, various works have sought to design fixed and quantization-focused optimization techniques that minimize induced performance degradation. However, there is little concrete understanding how decisions/best practices affect quantized inference behaviour. Weight initialization strategies are often associated with solving issues such as vanishing/exploding gradients but often-overlooked aspect their impact on final trained distributions each layer. We present in-depth, fine-grained ablation study effect different weights initializations activations architectures. The fine-grained, layerwise analysis enables us gain insights initial will accuracy To our best knowledge, we first perform a low-level, in-depth quantitative its